Cap a failed observation's error_message and reserve the openarmature_ namespace - #288
Merged
chris-colinsky merged 2 commits intoSep 3, 2026
Merged
Conversation
Implements proposal 0119, accepted at spec v0.116.0. Two halves. The cap. A failed observation's error_message now goes through the section 5.5.5 per-value byte cap at all four mapped provider observations. It takes the direct-application arm rather than the inheritance arm: the OTel surface defines no error_message span attribute, so the value arrives untruncated and the observer that writes it applies its own payload_byte_cap. Re-applying a second cap to an already-truncated value would move the marker and misreport its byte total, which is why the two arms source the cap differently. A failed Tool observation renders the same harvested string twice, in metadata.error_message and as the observation's status message, and both copies are capped. Section 5.5.5 governs payload-classified values rather than payload-classified fields, so capping one surface while the other carried the whole exception would defeat the cap. The remaining status message writes take the error category, a classification token, and stay uncapped. The omission arm is unchanged: a withheld message still leaves the status message null on a Tool failure. The reserved keys. openarmature_ joins openarmature. and gen_ai. as a reserved caller-metadata namespace prefix, and four exact names join the reserved set, taking it from 29 to 33. With error_message absent under the default posture, an unreserved caller key of that name would otherwise land unopposed in the very field 0118 requires to be absent. This is a pre-1.0 behavioral change: a caller passing invocation metadata with such a key is now rejected at the invoke boundary. Spec v0.116.0 is beyond the current v0.112.0 pin, so the behavior ships ahead of the pin, unit-tested; the conformance entry and fixtures ride the pin bump. No fixture can drive the Tool arm today because the conformance adapter exposes no tool-calling node.
There was a problem hiding this comment.
Pull request overview
Implements accepted spec proposal 0119 (v0.116.0) in the Python implementation by (1) enforcing the per-value byte cap on failed-observation error_message in the Langfuse mapping and (2) expanding reserved caller invocation-metadata keys to prevent collisions/leaks under the default privacy posture.
Changes:
- Cap
error_messageemission in the Langfuse observer (LLM, tool, embedding, rerank), including the tool arm’s duplicatedstatusMessagesurface. - Reserve the
openarmature_invocation-metadata namespace prefix and add new reserved exact keys (error_type,error_message,token_budget,token_budget_exceeded). - Add focused unit tests and update docs/changelog to reflect the new behavior.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
tests/unit/test_observability_metadata.py |
Adds unit tests asserting openarmature_ prefix and new reserved exact keys are rejected at validation. |
tests/unit/test_observability_langfuse.py |
Adds unit tests asserting error_message truncation behavior across observer arms and tool status_message. |
src/openarmature/observability/metadata.py |
Expands reserved prefixes and reserved key names for invocation metadata validation. |
src/openarmature/observability/langfuse/observer.py |
Applies payload_byte_cap truncation to error_message emission, including tool status-message duplication. |
src/openarmature/AGENTS.md |
Updates agent-facing guidance to reflect capped error_message behavior under Langfuse payload settings. |
docs/concepts/observability.md |
Updates observability docs to describe capped error_message behavior and tool dual-surface capping. |
docs/agent/non-obvious-shapes.md |
Mirrors the Langfuse isolation/payload guidance updates for agent documentation. |
CHANGELOG.md |
Adds a release note entry describing the cap and reserved-namespace behavior (ships ahead of the spec pin). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Ten findings. The two largest were one defect seen from both sides. 0119 requires a repository check that fails when a top-level metadata key written by the section 8 mapping is neither reserved by name nor covered by a reserved namespace, and it was not built. It is now. It discovers the metadata bags from what is passed to a metadata argument rather than naming one: the observer builds five differently-named bags, so a check hardcoded to the obvious one would have scanned one of five and reported clean. That is the same subset failure the proposal warns about, one level down. The check finds three uncovered keys, all on the failure-isolation marker span, where caller metadata is merged last with no collision check so a caller key wins. They are held in a named exclusion set rather than reserved unilaterally: that span is a graph mechanism no mapping table covers, so whether the reserved set should reach an unmapped span is a spec question. A new unreserved key still fails the check, and a paired assertion fails if a held key stops being emitted so the exclusion cannot outlive its subject. The cap tests were vacuous. They asserted the value got shorter, not that it was truncated by the specified algorithm, so a marker-less byte chop passed all of them. They now pin the marker, its pre-truncation byte total, and that the kept bytes prefix the original. A multibyte case covers the code-point backtracking, which no test reached: at a 256-byte cap the cut lands on a 4-byte boundary and the loop is dead code, so the test uses 257. Truncation is now surrogate-safe. The cap runs on the failure path before the observation is created, and an observer that raises is only warned about, so an unguarded encode of harvested exception text would have deleted the observation reporting the failure. Also: reserved-key docs brought in sync across three sites and the module docstring, the prefix rejection message built from the tuple instead of a stale hand-copied list, a spec reference moved out of a docstring, a comment this change made stale removed, and test matchers tightened onto the rule rather than the echoed key.
chris-colinsky
deleted the
feature/0119-error-message-cap-and-reserved-keys
branch
September 3, 2026 05:40
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements proposal 0119, accepted at spec v0.116.0. It closes the two error-channel edges 0118 left open.
The cap
A failed observation's
error_messageis now subject to the §5.5.5 per-value byte cap on all four mapped provider observations (Generation, Embedding, Tool, Retriever). It was written verbatim before this: 0118 classified the field for gating without saying it was subject to truncation, so a provider returning a very large exception string (an echoed HTML error page) rendered in full.It takes the contract's direct-application arm rather than its inheritance arm. The OTel surface defines no
error_messagespan attribute, so the value arrives untruncated and the observer that writes it applies its ownpayload_byte_cap, rather than inheriting a cap from an upstream OTel truncation that never happened. Re-applying a second cap to an already-truncated value would move the marker and misreport its byte total, which is why the two arms source the cap differently.Both surfaces of the Tool arm are capped. A failed Tool observation renders the same harvested string twice, once in
metadata.error_messageand once as the observation'sstatusMessage. §5.5.5 governs payload-classified values rather than payload-classified fields, so capping one while the other still carried the whole exception would defeat the cap. Spec confirmed this reading after the work was done: an implementation that caps only the metadata copy is non-conforming, and §8.7 gains a sentence namingstatusMessageas the case in point. The remainingstatusMessagewrites take the error category, a classification token, and stay uncapped.The omission arm is unchanged. §6's prohibition on surfacing the message through
statusMessageis scoped to substituting for an omitted message, so a withheld message still leavesstatusMessagenull on a Tool failure. Under the default posture (disable_provider_payload=True) the field is absent entirely per 0118, so the cap is observable only where payloads are enabled.The reserved keys
openarmature_joinsopenarmature.andgen_ai.as a reserved caller-metadata namespace prefix, and four exact names (error_type,error_message,token_budget,token_budget_exceeded) join the reserved set, which grows from 29 to 33.With
error_messageabsent under the default posture, an unreserved caller key of that name would otherwise land unopposed in the very field 0118 requires to be absent, reintroducing through the metadata channel the leak the gate closes.Pre-1.0 behavioral change: a caller passing
invocation_metadatawith a key beginningopenarmature_, or with any of those four names, is now rejected at theinvoke()boundary withValueErrorwhere it previously passed.Testing
Seven cases across six tests, each mutation-verified. Eight mutants run, all killed.
Two of them mattered. Reverting the cap at each of the four sites individually left the embedding and rerank arms green: the first pass covered only the LLM and Tool handlers, and a passthrough mutation of the shared helper (which breaks all four sites at once) produced exactly two failures rather than four. The suite would have accepted a two-arm implementation of a four-arm rule. Both arms now have tests, each verified to kill only its own case, and the passthrough mutant now trips all four.
Ahead of the pin
Spec v0.116.0 is beyond the current v0.112.0 pin, so the behavior ships unit-tested and the
conformance.tomlentry and fixtures ride the pin bump, matching the precedent in this cycle.No fixture can drive the Tool arm today: the conformance adapter exposes no tool-calling node, so that arm is unit-tested only. Reported to spec rather than filing a fixture that would be deferred on arrival.Correction (2026-09-03): the struck sentence is wrong. The conformance adapter has driven tool calls since 0092:
calls_toolwith amock_toolblock is a supported directive,mock_tool: {raises: {error_type, message}}produces aToolCallFailedEventcarrying the fixture's own message, and fixture 098 case 2 already drives exactly that into a Langfuse Tool observation and assertserror_messagein its metadata. It runs green. The real gap is narrower: fixture 160 carries no Tool case, and one is writable today against existing machinery. Corrected in the CHANGELOG and retracted to spec. Fixture 160's other five cases span Embedding, Generation, and Retriever, and would have caught the two-arm gap independently; they land at the pin bump.